Definition

Consider a hypothetical probability distribution QQ and true probability distribution PP on the same sample space. Then, the Kullback-Leibler divergence (KL divergence or relative entropy),

DKL(P∥Q)=∑x∈𝒳P(x)log⁡P(x)Q(x).{\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in {\mathcal {X}}}P(x)\,\log {\frac {P(x)}{Q(x)}}{\text{.}}}

References

  1. https://en.wikipedia.org/wiki/Kullback–Leibler_divergence
  2. https://www.weizmann.ac.il/complex/falkovich/sites/complex.falkovich/files/uploads/PNI23.pdf